Integration of Voice Commands into a Virtual Reality Environment for Assembly Design

This paper addresses integration of voice commands within immersive virtual reality sessions. While wands, 3D mice, etc., are extensively used for navigation and other interaction in 3D environments, voice commands can provide a much more natural means of interaction, especially in applications in which the immersed user’s hands and vision may be otherwise occupied. The IBM WebSphere Voice Server SDK has been used to develop a VoiceXML application for recognition of voice commands issued by the immersed user. The Resin web server is used to broadcast these commands on a TCP port. A client written using the JackScript API for Jack reads the commands and executes corresponding functions. A comprehensive set of voice commands has been developed to provide significant functionality for assembly and maintenance simulations. One of the most important applications is a method by which the immersed human can walk and fly through the virtual environment, thereby greatly expanding the envelope of the virtual environment navigable under the constraints imposed by the limited tracking volume. Voice commands are also used to switch between different modes of interaction (such as grasping between fingers as opposed to using the palm), for capturing specific postures, and for replaying captured posture sequences so the immersed user can evaluate them. Voice commands allow one person to control the virtual environment while immersed in the environment, thereby improving productivity.